We construct a "ghost" point: x Section 4.5 of [4], we have From Lemma 3.1 and Proposition 3.2 in [48], we have null[ x ] The last relationship we want to show is just equation (13). We separate the discussion into deterministic and stochastic settings. The total complexity is then K T . By Corollary 3.2 and discussion in Section 3.2, Algorithm 1 combined with By Corollary 3.2, Algorithm 1 combined with EG/OGDA can solve such auxiliary We implement these algorithms in the same way as in Section 5. 17 (a) Distance to limit point We compare EG and Catalyst-EG under same stepsizes in Figure 4(a).

Then, we have that 1. y Fact 2. Let z:= [ x; y ] and z This can be easily proven using the AM-GM inequality. Fact 3. Let z:= [ x; y ] R It is a crucial building block for the algorithms in this work. The following classical theorem holds for AGD. We will start by giving a precise statement of Algorithm 1.Algorithm 1 Alternating Best Response (ABR)Require: g (,), Initial point z The basic idea is the following. The following two lemmas about the inexact APP A algorithm follow from the proof of Theorem 4.1 [ Here we provide their proofs for completeness.

Then, we have that 1. y Fact 2. Let z:= [ x; y ] and z This can be easily proven using the AM-GM inequality. Fact 3. Let z:= [ x; y ] R It is a crucial building block for the algorithms in this work. The following classical theorem holds for AGD. We will start by giving a precise statement of Algorithm 1.Algorithm 1 Alternating Best Response (ABR)Require: g (,), Initial point z The basic idea is the following. The following two lemmas about the inexact APP A algorithm follow from the proof of Theorem 4.1 [ Here we provide their proofs for completeness.

We are particularly interested in the setting where neither ex plicit knowledge about f,g are available nor their unbiased stochastic derivatives. In this zeroth-order setting, we assume that only noisy evaluations of f and g are available upon query to an oracle. The BLP problem was first introduced by Bracken and McGill in t he 1970s [7] followed by a more general form of problem involving joint constraints of outer and inner variables. This is a fundamental problem in engineering and economics with dire ct applications in problems such as decision making [48], supply chain [61, 59], network design [51, 43], transportation and planning [16, 83], and optimal design [4, 32]. More recently, BLP has f ound applications in many areas of machine learning and artificial intelligence. Zeroth-order methods apply to many optimization problems ( including the BLP) where for various reasons such as complexity, lack of access to an accurat e model, or computational limitations, there is no or limited access to the objective gradient.

The Hessian of neural networks can be decomposed into a sum of two matrices: (i) the positive semidefinite generalized Gauss-Newton matrix G, and (ii) the matrix H containing negative eigenvalues. We observe that for wider networks, minimizing the loss with the gradient descent optimization maneuvers through surfaces of positive curvatures at the start and end of training, and close to zero curvatures in between. In other words, it seems that during crucial parts of the training process, the Hessian in wide networks is dominated by the component G. To explain this phenomenon, we show that when initialized using common methodologies, the gradients of over-parameterized networks are approximately orthogonal to H, such that the curvature of the loss surface is strictly positive in the direction of the gradient.